Skip to content

[Custom Transactions] Add TxBuilder::get_next_commitment_stats #3921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

tankyleo
Copy link
Contributor

@tankyleo tankyleo commented Jul 10, 2025

    Add `TxBuilder::get_next_commitment_stats`

    Given a snapshot of the lightning state machine,
    `TxBuilder::get_next_commitment_stats` calculates the transaction fees,
    the dust exposure, and the holder and counterparty balances
    (the balances themselves do *not* account for the transaction fee).

and

    Add `ChannelContext::get_next_{local, remote}_commitment_stats`

    In upcoming commits, these methods will serve as proxies to
    `SpecTxBuilder::get_next_commitment_stats` in all validation of channel
    updates in `ChannelContext`.

    Eventually, these methods will completely replace
    `get_pending_htlc_stats`, and
    `get_next_{local, remote}_commit_tx_fee_msat`.

    When predicting the HTLCs on next commitment, we take the conservative
    approach and only assume that a HTLC will not be in the next commitment
    when it is guaranteed that it won't be.

@ldk-reviews-bot
Copy link

ldk-reviews-bot commented Jul 10, 2025

👋 Thanks for assigning @carlaKC as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleo tankyleo self-assigned this Jul 10, 2025
@tankyleo tankyleo moved this to Goal: Merge in Weekly Goals Jul 10, 2025
@tankyleo
Copy link
Contributor Author

tankyleo commented Jul 10, 2025

We discussed earlier passing the entire list of HTLCs to TxBuilder, and letting it do the dust-vs-non-dust sorting itself. There are nonetheless multiple places in channel that are interested in knowing the exact HTLC dust limit (see get_pending_{inbound, outbound}_htlc_details, get_available_balances_for_scope).

So we prefer to add a single method that surfaces this limit to channel, and then let channel sort HTLCs depending on whether their value sits above or below that amount.

As a result, builders of custom transactions will only have a say on where the dust limit sits, and won't be able to choose arbitrary subsets for dust and non-dust HTLCs.

@carlaKC carlaKC self-requested a review July 10, 2025 17:43
@carlaKC
Copy link
Contributor

carlaKC commented Jul 10, 2025

We discussed earlier passing the entire list of HTLCs to TxBuilder, and letting it do the dust-vs-non-dust sorting itself.

Is the main motivation for this to get all of the commitment-related logic out of channel.rs, or that we think that custom tx builders will want to specifically choose certain htlcs to be dust?

get_available_balances_for_scope

The key questions we're looking to answer seems to be "can I afford a commitment with this theoretical dust/nondust htlc"? This does seem to be something we could move into TxBuilder if we pass the full HTLC set and use a version of HTLCCandidate which just tells TxBuilder whether it is dust (we never actually use this amount other than the dust check).

Sadly here's no getting around needing to know the dust limit if we want to clamp our capacity to that value, so we'd still need to surface htlc_success_timeout_dust_limits.

get_pending_{inbound, outbound}_htlc_details

Tempting to suggest just adding an is_dust check to TxBuilder for these to make it slightly more abstract than surfacing the second stage fees, but given the above requirement to know the exact dust limit maybe we just want to leave as-is.

so tl;dr: I'd be interested in seeing what trying to pull more of the dust logic out into TxBuilder looks like, even if just gives us a clearer idea of where the trait line should be.


Meta note: This has got a lot of overlap with 3bb0586, so I think we should either:

  1. Rebase Update fee and dust handling for zero fee channels #3884 on this PR
  2. Pull the commit out into a common prefactor (I have a slight pref for this, but happy with either)

@tankyleo
Copy link
Contributor Author

Is the main motivation for this to #3775 (comment), or that we think that custom tx builders will want to specifically choose certain htlcs to be dust?

I would say we focus on the latter for this PR, and let the former be the overarching goal :)

so tl;dr: I'd be interested in seeing what trying to pull more of the dust logic out into TxBuilder looks like, even if just gives us a clearer idea of where the trait line should be.

Let me know what you think of this new direction here. Still have some clunkiness to resolve, but that's what it's looking like right now.

@tankyleo
Copy link
Contributor Author

Meta note: This has got a lot of overlap with 3bb0586

Definitely let me rebase on top of your PR, you've rebased once already :)

@tankyleo tankyleo force-pushed the dust-limit branch 2 times, most recently from 12fd6c3 to 067bfa0 Compare July 11, 2025 09:41
@ldk-reviews-bot
Copy link

🔔 1st Reminder

Hey @carlaKC! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@carlaKC
Copy link
Contributor

carlaKC commented Jul 15, 2025

Let me know what you think of this new direction here.

I like this approach! Definitely prefer being able to move a lot of the dust / second stage tx reasoning into the builder 👍

@ldk-reviews-bot
Copy link

🔔 2nd Reminder

Hey @carlaKC! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleo tankyleo removed the request for review from carlaKC July 18, 2025 18:01
@tankyleo tankyleo changed the title [Custom Transactions] Abstract the weight of HTLC transactions away from channel [Custom Transactions] Abstract dust-vs-nondust HTLC sorting away from channel Jul 21, 2025
@tankyleo tankyleo marked this pull request as ready for review July 21, 2025 08:56
@tankyleo tankyleo requested a review from TheBlueMatt July 21, 2025 08:56
@tankyleo
Copy link
Contributor Author

I'm aware I still owe a follow-up PR from the previous PR in this project, will push that soon in a separate PR :)

Comment on lines 42 to 46
fn on_holder_tx_dust_exposure_msat(
&self, dust_buffer_feerate: u32, holder_dust_limit_satoshis: u64,
channel_type: &ChannelTypeFeatures, htlcs: &[HTLCAmountDirection],
) -> u64;
fn on_counterparty_tx_dust_exposure_msat(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really sure I understand the point of these two methods if we have htlc_success_timeout_dust_limit as a method - if the API includes a method to say "hey, for an HTLC of type X what is the threshold where its dust", why bother with a method to say "given these HTLCs, how many are dust?", it seems the second can be calculated from the first, no? The same applies for passing the HTLC list to commit_tx_fee_sat.

Alternatively, we could drop the htlc_success_timeout_dust_limits method (if we want to not require there be some strict threshold to indicate when an HTLC is dust, though I think its a fine assumption to bake into the API, I dunno why someone would want to have some HTLCs be dust and others at the same value not be) and have a more generic "is HTLC dust on the next counterparty/local commitment transaction" call, but even there it seems like we don't need all of these methods.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good yes I initially had a version of the API that matches what you describe. We wanted to see what it would look like if we moved more logic out of channel into TxBuilder, but I agree we only really need the dust limits.

I will clean up those methods, and add TxBuilder methods for 1) the dust limit 2) the htlc endogenous fees (to calculate counterparty dust exposure).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean don't get me wrong, I'd love to move more logic out of channel.rs into TxBuilder, that would be great, but we also don't want to have an overlapping API where we have two ways to do things over the API. Also, we want to eventually make TxBuilder public, so it would be nice to avoid adding too much complexity (in the form of a ton of different methods).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW the real blocker on the more ambitious / complex API was the get_available_balances method - at the moment this method requires an exact dust limit above which all HTLCs are non-dust.

I have some code written that moves most of get_available_balances behind TxBuilder, but the complexity is not worth it, especially if we can make the assumption you describe above about the strict dust-vs-nondust threshold.

@ldk-reviews-bot
Copy link

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

@tankyleo tankyleo changed the title [Custom Transactions] Abstract dust-vs-nondust HTLC sorting away from channel [Custom Transactions] Let TxBuilder set the HTLC dust limit Jul 22, 2025
@tankyleo tankyleo changed the title [Custom Transactions] Let TxBuilder set the HTLC dust limit [Custom Transactions] Let TxBuilder set the HTLC dust limit Jul 22, 2025
Copy link

codecov bot commented Jul 22, 2025

Codecov Report

❌ Patch coverage is 97.75281% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.85%. Comparing base (39e8d7d) to head (b475944).
⚠️ Report is 138 commits behind head on main.

Files with missing lines Patch % Lines
lightning/src/ln/channel.rs 96.31% 7 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3921      +/-   ##
==========================================
- Coverage   88.93%   88.85%   -0.09%     
==========================================
  Files         174      175       +1     
  Lines      123876   127872    +3996     
  Branches   123876   127872    +3996     
==========================================
+ Hits       110173   113623    +3450     
- Misses      11250    11686     +436     
- Partials     2453     2563     +110     
Flag Coverage Δ
fuzzing 22.10% <85.11%> (-0.08%) ⬇️
tests 88.68% <97.75%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@elnosh elnosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for what it can be worth it I have reviewed 1eb1c9e and it looks correct to me.

As for my understanding on the reasoning for the change, motivation seems to be to move usage of second_stage_tx_fees_sat and htlc_tx_fees_sat to TxBuilder in order to abstract away usage of weight of htlc transactions away from channel.rs.

@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 4, 2025

Rebase: diff

  • Update comments,
  • Read pending inbound htlcs count and value from TxBuilder,
  • Use include_counterparty_unknown_htlcs parameter to count outbound update_add_htlcs in the holding cell, and those not yet ACK'ed by the counterparty.

@tankyleo tankyleo force-pushed the dust-limit branch 2 times, most recently from 0904e98 to 3b851ba Compare August 5, 2025 07:54
@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 5, 2025

Rebase: full diff

  • squash
  • nits
  • look at the next remote commitment stats in can_send_update_fee, not the next local commitment stats.
  • Expand git messages

@tankyleo tankyleo requested a review from carlaKC August 5, 2025 08:41
@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 5, 2025

@carlaKC I believe the PR is ready for a deeper look :) Can't quite figure out why CI fails to build, it does pass on my machine.

I thought we could focus on the first two commits for now, and we push the validation / prediction commits to the next PR in this series.

Let me know what you'd prefer.

I've pushed the validation / prediction commits here too so you have the context. Also see their commit messages for further description of the changes.

@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 6, 2025

@carlaKC A correction: In the b16779a commit message, I say "This commit also includes outbound HTLC additions in the holding cell in the fees on the next remote commitment transaction; they were previously not included."

This is wrong, they are already included via the additional HTLC parameter of ChannelContext::build_commitment_stats, will make the correction.

@ldk-reviews-bot
Copy link

🔔 1st Reminder

Hey @carlaKC! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 7, 2025

I also plan to expand the commit messages to describe in detail how we now read the feerates we use to calculate the next commitment stats.

Copy link
Contributor

@carlaKC carlaKC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed first two commits - API looks good!

I don't have strong feelings on how to split this up - happy to separate API/usage or do both in one go. @tankyleo lmk your preference and I'll review accordingly.

I'm ready for this to get a rebase/squash so it can pick up some minor relevant V3 channel changes.

/// We take the conservative approach and only assume that a HTLC will
/// not be in the next commitment when it is guaranteed that it won't be.
#[allow(dead_code)]
#[rustfmt::skip]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: don't skip formatting on new code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On my machine rustfmt is not consistent when formatting get_next_commitment_htlcs and get_next_commitment_value_to_self_msat, and also makes a big mess of get_next_commitment_value_to_self_msat :)

Given a snapshot of the lightning state machine,
`TxBuilder::get_next_commitment_stats` calculates the transaction fees,
the dust exposure, and the holder and counterparty balances
(the balances themselves do *not* account for the transaction fee).
In upcoming commits, these methods will serve as proxies to
`SpecTxBuilder::get_next_commitment_stats` in all validation of channel
updates in `ChannelContext`.

Eventually, these methods will completely replace
`get_pending_htlc_stats`, and
`get_next_{local, remote}_commit_tx_fee_msat`.

When predicting the HTLCs on next commitment, we take the conservative
approach and only assume that a HTLC will not be in the next commitment
when it is guaranteed that it won't be.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs
on the next commitment will be all the HTLCs in
`ChannelContext.pending_inbound_htlcs`, and
`ChannelContext.pending_outbound_htlcs`, as well as all the outbound
HTLC adds in the holding cell.

This is an overestimate:

* Outbound HTLC removals which have been ACK'ed by the counterparty will
  certainly not be present in any *next* commitment, even though they
  remain in `pending_outbound_htlcs`.

* Outbound HTLCs in the `RemoteRemoved` state, will not be present in
  the next *local* commitment.

* Outbound HTLCs in the `LocalAnnounced` state have no guarantee that
  they were received by the counterparty before she sent the
  `update_fee`.

* Outbound `update_add_htlc`'s in the holding cell are certainly not
  known by the counterparty, and we will reevaluate their addition to
  the channel when freeing the holding cell.

* Inbound HTLCs in the `LocalRemoved` state will not be present in the
  next *remote* commitment.

`ChannelContext::next_local_commit_tx_fee_msat` over-counts outbound
HTLCs in the `LocalAnnounced` and `RemoteRemoved` states, as well as
outbound `update_add_htlc`'s in the holding cell.

`ChannelContext::next_remote_commit_tx_fee_msat` over-counts inbound
HTLCs in the `LocalRemoved` state, as well as outbound HTLCs in the
`LocalAnnounced` state.

This commit stops using these functions in favor of the newly added
`ChannelContext::get_next_{local, remote}_commitment_stats` methods,
and fixes the issues described above.

If we are the funder, we also check that adding this inbound HTLC
doesn't increase the commitment transaction fee to the point of
exhausting our balance on the local commitment. Previously, we would
only subtract the anchors from `funding.value_to_self_msat`; we now
also subtract the outbound HTLCs on the next local commitment from
`funding.value_to_self_msat` before checking if we can afford the
additional transaction fees.

Inbound `LocalRemoved` HTLCs that were **not** successful are now
credited to `remote_balance_before_fee_msat` as they will certainly not
be on the next remote commitment. We previously debited these from the
remote balance to arrive at `remote_balance_before_fee_msat`.

When calculating dust exposure, we now take a buffer from the currently
committed feerate, and ignore any fee updates in
`ChannelContext.pending_update_fee`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs
on the next commitment will be all the HTLCs in
`ChannelContext.pending_inbound_htlcs`, and
`ChannelContext.pending_outbound_htlcs`, as well as all the outbound
HTLC adds in the holding cell.

This is an overestimate:

* Outbound HTLC removals which have been ACK'ed by the counterparty will
  certainly not be present in any *next* commitment, even though they
  remain in `pending_outbound_htlcs`.

* Outbound HTLCs in the `RemoteRemoved` state, will not be present in
  the next *local* commitment.

* Outbound HTLCs in the `LocalAnnounced` state have no guarantee that
  they were yet received by the counterparty.

* Outbound `update_add_htlc`'s in the holding cell are certainly not
  known by the counterparty, and we will reevaluate their addition to
  the channel when freeing the holding cell.

* Inbound HTLCs in the `LocalRemoved` state will not be present in the
  next *remote* commitment.

This commit stops using `get_pending_htlc_stats` in favor of the newly
added `ChannelContext::get_next_{local, remote}_commitment_stats`
methods, and fixes the issues described above.

`ChannelContext::next_remote_commit_tx_fee_msat` counts inbound HTLCs in
the `LocalRemoved` state, as well as outbound HTLCs in the
`LocalAnnounced` state. We now do not count them for the same reasons
described above.

Inbound `LocalRemoved` HTLCs that were **not** successful are now
credited to `remote_balance_before_fee_msat` as they will certainly not
be on the next remote commitment. We previously debited these from the
remote balance to arrive at `remote_balance_before_fee_msat`.

We now always check holder dust exposure, whereas we previously would
only do it if the incoming HTLC was dust on our own commitment
transaction.

Furthermore, dust exposure calculations now take a buffer from the
currently committed feerate, and ignore any fee updates in
`ChannelContext.pending_update_fee`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs
on the next commitment will be all the HTLCs in
`ChannelContext.pending_inbound_htlcs`, and
`ChannelContext.pending_outbound_htlcs`, as well as all the outbound
HTLC adds in the holding cell.

This is an overestimate:

* Outbound HTLC removals which have been ACK'ed by the counterparty will
  certainly not be present in any *next* commitment, even though they
  remain in `pending_outbound_htlcs`.

* Outbound HTLCs in the `RemoteRemoved` state, will not be present in
  the next *local* commitment.

* Outbound HTLCs in the `LocalAnnounced` state have no guarantee that
  they were received by the counterparty before she sent the
  `update_fee`.

* Outbound `update_add_htlc`'s in the holding cell are certainly not
  known by the counterparty, and we will reevaluate their addition to
  the channel when freeing the holding cell.

* Inbound HTLCs in the `LocalRemoved` state will not be present in the
  next *remote* commitment.

This commit stops using `get_pending_htlc_stats` in favor of the newly
added `ChannelContext::get_next_{local, remote}_commitment_stats`
methods, and fixes the issues described above.

We now always calculate dust exposure using a buffer from
`msg.feerate_per_kw`, and not from
`max(self.feerate_per_kw, msg.feerate_per_kw)`.
`ChannelContext::get_pending_htlc_stats` predicts that the set of HTLCs
on the next commitment will be all the HTLCs in
`ChannelContext.pending_inbound_htlcs`, and
`ChannelContext.pending_outbound_htlcs`, as well as all the outbound
HTLC adds in the holding cell.

This is an overestimate:

* Outbound HTLC removals which have been ACK'ed by the counterparty will
  certainly not be present in any *next* commitment, even though they
  remain in `pending_outbound_htlcs` (I refer to states
  `AwaitingRemoteRevokeToRemove` and `AwaitingRemovedRemoteRevoke`).

* Outbound HTLCs in the `RemoteRemoved` state, will not be present in
  the next *local* commitment.

* Inbound HTLCs in the `LocalRemoved` state will not be present in the
  next *remote* commitment.

`ChannelContext::build_commitment_stats(funding, true, true, ..)` makes
these errors when predicting the HTLC count on the remote commitment:

* Inbound HTLCs in the state `RemoteAnnounced` are not included, but
  they will be in the next remote commitment transaction if the local
  ACK's the addition before producing the next remote commitment.

* Inbound HTLCs in the state `AwaitingRemoteRevokeToAnnounce` are not
  included, even though the local has ACK'ed the addition.

* Outbound HTLCs in the state `AwaitingRemoteRevokeToRemove` are
  counted, even though the local party has ACK'ed the removal.

This commit replaces these functions in favor of the newly added
`ChannelContext::get_next_{local, remote}_commitment_stats` methods,
and fixes the issues described above.

We now always calculate dust exposure using a buffer from
`msg.feerate_per_kw`, and not from
`max(feerate_per_kw, self.feerate_per_kw, self.pending_update_fee)`.
Anytime we build a (feerate, nondust-htlc-count, fee) pair, cache it,
and check that the fee matches if the feerate and nondust-htlc-count
match when building a commitment transaction.
The cached fee is never checked in the current test suite.
@tankyleo
Copy link
Contributor Author

tankyleo commented Aug 11, 2025

Rebase: squash, nits, edit commit messages; full diff

@tankyleo tankyleo requested a review from carlaKC August 11, 2025 07:23
@ldk-reviews-bot
Copy link

🔔 1st Reminder

Hey @carlaKC! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

let include_counterparty_unknown_htlcs = false;
// Don't include the extra fee spike buffer HTLC in calculations
let fee_spike_buffer_htlc = 0;
let next_remote_commitment_stats = self.get_next_remote_commitment_stats(funding, Some(HTLCAmountDirection { outbound: false, amount_msat: msg.amount_msat }), include_counterparty_unknown_htlcs, fee_spike_buffer_htlc, self.feerate_per_kw, dust_exposure_limiting_feerate);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we hit the following here?

  • Remote peer sends an update_add_htlc for an amount > their current balance
  • get_next_remote_commitment_stats includes the candidate in next_commitment_htlcs (get_next_commitment_htlcs always adds the candidate)
  • get_next_commitment_stats includes the large htlc in inbound_htlcs_value_msat and the checked_sub on value_to_counterparty_msat's expect fails

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

woah thank you ! yes will be sure to fix this :)

Do not panic if the counterparty adds a HTLC with a value greater than
its remaining balance!
@tankyleo tankyleo requested a review from carlaKC August 13, 2025 17:28
For consistency, include the number of additional nondust HTLCs in the
total nondust HTLC count; commit_tx_fee_sat also includes these.
@ldk-reviews-bot
Copy link

🔔 1st Reminder

Hey @carlaKC! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Goal: Merge
Development

Successfully merging this pull request may close these issues.

5 participants